{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 18\n",
    "---\n",
    "# Naive Bayes\n",
    "\n",
    "### 18.0 Introduction\n",
    "Bayes' theorem is the premier method for understanding the probability of some event $P(A|B)$, given some new information, $P(B|A)$, and a prior belief in the probability of the event, P(A):\n",
    "$$\n",
    "P(A | B) = \\frac{P(B|A)P(A)}{P(B)}\n",
    "$$\n",
    "\n",
    "The Bayesian method's popularity has skyrocked in the last decade, more and more rivaling the traditional frequentist applications in academia, government, and business. In machine learning, one applicaiton of Bayes' theorem to classifican comes in the form of the naive Bayes classifier. Naive Bayes classifiers combine a number of desirable qualities in practical machine learning into a single classifier:\n",
    "\n",
    "1. An intuitive approach\n",
    "2. The ability to work with small data\n",
    "3. Low computation costs for training and prediction\n",
    "4. Often solid results in a variety of settigns\n",
    "\n",
    "Specifically, a naive bayes classifier is based on:\n",
    "$$\n",
    "P(y | x_1, ..., x_j) = \\frac{P(x_1, ..., x_j | y)P(y)}{P(x_1,...,x_j)}\n",
    "$$\n",
    "where,\n",
    "* $P(y | x_1, ..., x_j)$ is called the *posterior* and is the probability that an observation is class y given observation's values for the j features, $x_1, ..., x_j$\n",
    "* $P(x_1, ..., x_j)$ is called likelihood and is the *likelihood* of an observation's values for features, $x_1, ..., x_j$, given their class y.\n",
    "* $P(y)$ is called the *prior* and is our belief for the probability of class y before looking at the data\n",
    "* P($x_1, ..., x_j$) is called the *marginal probability*\n",
    "\n",
    "In naive Bayes, we compare an obsrvation's posterior values for each possible class. Specifically, because the marginal probability is constant across these comparisons, we compare the numerators of the posterior for each class. For each observation the class with the greatest posterior numerator becomes the predicted class, $\\hat y$.\n",
    "\n",
    "There are two important things to note about naive Bayes classifiers.\n",
    "\n",
    "1. for each feature in the data, we have to assume the statistical distribution of the likelihood, $P(x_1, ..., x_j)$.\n",
    "- the common distributions are the normal (Gaussian), multinomial, and Bernoulli distributions.\n",
    "- the distribution chose is often determined by the nature of the features (continuous, binary, etc.)\n",
    "\n",
    "2. naive Bayes gets its name because we assume that each feature, and its resulting likelihood, is independent. This \"naive\" assumption is frequently wrong, yet in practice does little to prevent building high quality classifiers\n",
    "\n",
    "## 18.1 Training a Classifier for Continuous Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "features = iris.data\n",
    "target = iris.target\n",
    "\n",
    "classifier = GaussianNB()\n",
    "\n",
    "model = classifier.fit(features, target)\n",
    "\n",
    "new_observation = [[4, 4, 4, 0.4]]\n",
    "model.predict(new_observation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion\n",
    "The most common type of naive Bayes classifier is the Gaussian naive Bayesa. In Gaussian naive Bayesam we assuem that the likelihood of the feature values, x, given an observation is of class y, follows a normal distribution:\n",
    "$$\n",
    "p(x_j | y) = \\frac{1}{\\sqrt{2\\pi \\sigma_y^2}} e^{-\\frac{(x_j - \\mu_y)^2}{2\\sigma_y^2}}\n",
    "$$\n",
    "where $\\sigma_y^2$ and $\\mu_y$ are the variance and mean values of feature x_j for class y. Because of the assumption of the normal distribution, Gaussian naive Bayes is best used in cases when all our features are continuous.\n",
    "\n",
    "One of the interesting aspects of naive Bayes classifiers is that they allow us to assign a prior belief over the respect target classes. We can do this using `GaussianNB`'s `priors` parameter, which takes in a list of the probabilities assigned to each class of the target vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = GaussianNB(priors=[0.25, 0.25, 0.5])\n",
    "model = classifier.fit(features, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### See Also\n",
    "* How the Naive Bayes Classifier Works (http://dataaspirant.com/2017/02/06/naive-bayes-classifier-machine-learning/)\n",
    "\n",
    "## 18.2  Training a Classifier for Discrete and Count Features\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "text_data = np.array(['I love Brazil. Brazil!', 'Brazil is best', 'Germany beats both'])\n",
    "\n",
    "count = CountVectorizer()\n",
    "bag_of_words = count.fit_transform(text_data)\n",
    "\n",
    "features = bag_of_words.toarray()\n",
    "\n",
    "target = np.array([0, 0, 1])\n",
    "\n",
    "classifier = MultinomialNB(class_prior=[0.25, 0.5])\n",
    "model = classifier.fit(features, target)\n",
    "\n",
    "new_observation = [[0, 0, 0, 1, 0, 1, 0]]\n",
    "model.predict(new_observation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion\n",
    "\n",
    "Multinomial naive Bayes works similarly to Gaussian naive Bayes, but the features are assumed to be multinomial distributed. In practice this means that this classifier is commonly used when we have discrete data. One of the most common uses is text classification using bag of words or tf-idf approaches\n",
    "\n",
    "## 18.3 Training a Naive Bayes Classifier for Binary Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "\n",
    "features = np.random.randint(2, size=(100, 3))\n",
    "target = np.random.randint(2, size=(100, 1)).ravel()\n",
    "\n",
    "classifier = BernoulliNB(class_prior=[0.25, 0.5])\n",
    "model = classifier.fit(features, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Bernoulli naive Bayes classifier assumes that all our features are binary such that they take only two values (e.g. a nominal categorical feature that has been one-hot encoded). Like its multinomial cousin, Bernoulli naive Bayes is often used in text classification, when our feature matrix is simply the presence or absence of a word in a document"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:machine_learning_cookbook]",
   "language": "python",
   "name": "conda-env-machine_learning_cookbook-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
